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Image Display Method 

5 Technical Field 

The present invention relates to an image display method 
characterized in that an image, which is transmitted from a 
photographic device mounted onto, for example, a helicopter, 
is displayed being superposed .on a map of a geographic 
10 information system, thereby enabling to determine situations 1 
on the ground easily as well as with sufficient precision in 
the case where natural disaster such as earthquake or fire, 
or human disaster such as explosion or serious accident occur. 

15 Background Art 

In the conventional positional specification method and 
device, as shown, for example,, in the Japanese Patent No. 2695393, 
a shooting position in the air is specified three-dimensional ly , 
a direction of a target with respect to a shooting position 

20 is measured, a ground surface where the target ■ resides is 
obtained based on a three-dimensional topographic data 
including altitude information as to undulation of the ground, 
surface which data has been preliminarily prepared, and a 
position of the target on the ground surface having been shot 

25 from the air is specified as a point of intersection - of the 
ground surface with a straight line extending from the shooting 
position toward the target. 

In the conventional positional specification method and 
device, to specify the position of a target on the ground surface, 

30 a three-dimensional topographic data including altitude 
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information as to undulation of the ground surface which data 
has been preliminarily prepared is needed as a prerequisite.' 
Further, measurement error that occurs at the time of specifying 
three-dimensionally a shooting position in the air and at the 
5 time of measuring the direction of a target with respect to 
the shooing position cannot be compensated, thus making it hard 
to specify a position with accuracy. Furthermore, since the 
positional specification is executed with respect to one point 
of target, a problem exists in that situations on the ground 
10 surface cannot be understood area-wide. 

Disclosure of Invention 

The present invention was made to solve the above-discussed 
problems, and has an object of providing an image display method 
15 in which shot images are displayed being superposed on a map 
of a geographic information system, thereby enabling to 
understand area-wide situations on the ground surface having 
been shot; as well as in which a display position on the map 
of an image is compensated by comparison between the shot image 
and the map to carry out the superposed display with high 
precision, thereby enabling to understand situations of the 
ground surface having been shot more easily and rapidly. 

To accomplish the foregoing ob j ects , an image display method 
according to the invention is a method of image processing and 
25 displaying a shot image of the ground surface having been taken 
with photographic equipment that is mounted on an airframe in 
the air, in which a shooting position in the air is specified 
three-dimensionally, a photographic area on the ground surface 
having been shot is obtained by computation, and a shot image 
is transformed in conformity with the mentioned photographic 
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area and thereafter displayed being superposed on a map of a 
geographic information system. 

A further image display method is a method of image 
processing and displaying a shot image of the ground surface 
5 having been taken with photographic equipment that is mounted 
on an airframe in the air, in which a shooting position in the 
air is specified three-dimensionally , a photographic area of 
the ground surface having been shot is obtained by computation, 
and a shot image is transformed in conformity with the mentioned 

10 photographic area and thereafter displayed being superposed 
on a map of • a geographic information system; and in which 
landmarks are extracted from a map of the geographic information 
system and a shot image respectively and the corresponding- 
landmarks are compared, whereby a parameter for use in computing 

15 a photographic area of the ground surface having been shot is 
compensated, .and a shot image is displayed being superposed 
with high precision on a map of the geographic information 
system. 

According to this invention, it becomes easy to ascertain 
20 conformity between image information and a map, thereby enabling 
to identify a target. point of land easily. 

Brief Description of Drawings 

Fig. 1 is a block diagram showing a system for carrying 
25 out an image display method according to a first preferred, 
embodiment of the present invention. 

Fig. 2 is an explanatory diagram of functions of map 
processing means in the first embodiment. 

Fig. 3 is a photograph showing a display screen according 
30 to the first embodiment. 
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Fig. 4 is a photograph showing a display screen obtained 
by an image display method according to a second embodiment 
of the invention. 

Figs. 5 are views explaining a third embodiment of the 
invention . 

Figs . 6 are diagrams explaining map processing in the third 
embodiment . 

Figs. 7 are views explaining a fourth embodiment of the 
invention. 

Figs . 8 are diagrams explaining map processing in the fourth 
embodiment . 

Figs. 9 are views explaining a fifth embodiment of the 
invention. 

Figs . 10 are diagrams explaining map processing in the fifth 
embodiment . 

Fig. 11 is a diagram for explaining map processing of an 
image display method according to a sixth embodiment of the 
invention. 

Fig. 12 is a view for explaining map processing of an image 
display method according to a seventh embodiment of the 
invention. 

Figs. 13 are views explaining an image display method 
according to an eighth embodiment of the invention. 

Fig. 14 is a block diagram showing a system for carrying 
out an image display method according to a ninth embodiment 
of the invention. 

Fig. 15 is an explanatory diagram of functions of map. 
processing means in the ninth embodiment. 

Fig. 16 is a flowchart showing operations in the image 
display method according to the ninth embodiment. 



Figs. 17 are views explaining angle parameters for use in 
computing a photographic frame in map processing means according 
to the ninth embodiment . 

Figs. 18 are diagrams explaining the photographic frame 
5 computation in map processing means according to the ninth 
embodiment . 

Fig. 19 is a diagram explaining parameter compensation in 
map processing means according to the ninth embodiment. 

Figs. 20 are views showing effects in the image display 
10 method according to the ninth embodiment. 

Figs. 21 are views explaining an eleventh embodiment of 
the invention. 

Figs. 22 are diagrams explaining a twelfth embodiment of 
the invention. 

15 Fig. 23 is a flowchart showing operations in an image display 

method according to a fourteenth embodiment of the invention. 

Fig. 24 is a view showing effects in the image display method- 
according to the fourteenth embodiment. 

Figs. 25 are diagrams explaining a fifteenth embodiment 
20 of the invention. 

Fig. 26 is a view explaining a sixteenth embodiment of the 
invention. 

Best Mode for Carrying out the Invention 

2 5 Embodiment 1. 

First, the present invention is summarized. The invention 
is to display a shot image of the ground having been shot from 
the air, being superposed on a map of a geographic information 
system (GIS=Geographic Information System, system of. 

30 displaying a map on the computer screen) , thereby making it 
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easy to acknowledge conformity between an image information 
and a map, and making it easy to determine a target point of 
land. In this regard, in the case of taking a shot of the ground 
from the air with a camera, an image thereof is taken only in 
a certain rectangular shape at all times regardless of direction 
of the camera. Therefore, it is difficult to superpose (paste) 
as it is an image having been shot on a map that is obtained 
with the ■ geographic information system. Thus, according to 
this invention, a photographic area (=photographic frame) of 
the ground surface to be shot, the photographic area 
complicated! y varying from a rectangle to a shape close to 
trapezoid or rhombus depending on, e.g. , posture of the camera 
with respect to the ground, is obtained by computation using 
camera information and posture information of an airframe at 
the time of shooting an image . Then the shot image is transformed 
in conformity with the image frame, pasted onto the map, and 
displayed. 

Hereinafter, an image processing method and an image 
processing system according to a first preferred embodiment 
of the invention is described with reference to the drawings. 
Fig. 1 is a block diagram explaining a system of carrying out 
the method of the invention. Fig. 2 is a diagram explaining 
functions of map processing means . The method of the invention 
is implemented with an on-board system 100 formed of a flight 
vehicle (=airframe) such as helicopter on which, e.g., 
photographic equipment (=camera) is mounted, and a ground system 
200 located on the ground that receives signals from the on-board 
system 100 and processes them. 

The on-board system 100 is formed of on-board devices 
including photographic means for taking a shot of the ground 



from the air, airframe position measurement means 108 or airframe 
posture measurement means 107 acting as information collection 
section that obtains information for specifying 
three-dimensionally a shooting position of photographic means, 
5 and transmission means for transmitting a shot image having 
been taken by the mentioned photographic means and information 
obtained by the mentioned information collection section. 

More specifically, on the on-board system 100, a camera 
102 acting as photographic means 105 that takes pictures of 

10 the ground from the air is mounted. The airframe 101 is provided 
with airframe position measurement means 108 that obtains 
current positional information with an antenna 103, being a 
GPS signal receiving section, and detects an airframe position, 
and a gyro. The airframe 101 is further provided with airframe 

15 posture measurement means 107 that performs airframe posture 
detection of detecting a posture, that is, an elevation angle 
(=pitch) and a roll angle of the airframe 101. 

The photographic means 105 including the camera 102 takes 
a shot of the ground and outputs image signals thereof, and 

20 also outputs camera information such as diaphragm or zoom of 
the camera as well. The camera 102 is attached to a gimbal, 
and this gimbal includes camera posture measurement means 106 
detecting a rotation angle (=pan) and inclination (=tilt) of 
the camera, and outputs values thereof. 

25 An output signal from the mentioned airframe position 

measurement means 108, an output signal from the mentioned 
airframe posture measurement means 107, an image signal and 
a camera information signal of the mentioned camera shooting 
means 105, an output signal from the mentioned camera posture 

30 measurement means 106 are multiplexed and modulated by 



7 



multiplex modulator 109 . • These signals are converted to digital 
signals by signal conversion means 110, and transmitted to the 
ground system 200 from transmission means 104 having tracking 
means 111. 

5 The ground system 200 is mainly constituted of: an input 

section that inputs a shot image of the ground surface, which 
photographic means takes from the air, and information for 
three-dimensionally specifying a shooting position of the 
above-mentioned photographic means ; a signal processing 

10 section that performs signal processing with respect to 
information, having been inputted; a geographic information 
system that displays a map on the screen; and a map processing 
section that processes the image as well as the information 
having been processed at the signal processing section, and 

15 displays the resultant picture on the monitor. 

More specifically, signals from the on-board system 100 
are received with receiving means 201 including tracking means 
202, and signal-convertedby signal conver sionmeans 203 . These 
signals are fetched, out as image signals and the other 

20 information signals such as airframe position, airframe posture, 
camera posture or camera information with multiplex demodulator 
204. These fetched-out signals are signal-processed with 
signal processing means 205, and the image signals are used 
in map processing with map processing means 206 in the next 

25 step as a moving image data 207 and a still image data 208. 
Other information signals including a two-dimensional map data 
209 and a topographic data 210 of the geographic information 
system are also used in map processing with map processing means 
206. Numeral 211 designates monitor display means. 

30 Fig. 2 is a schematic diagram showing map processing means 



of the image display system according to this first embodiment . 
The map processing means 206, as shown in Fig. 2, executes the 
processing with a moving image data 207 and still image data 
208, being image signals, information signals of airframe 
5 position, airframe posture and camera posture, and a 
two-dimensional map data 209 and a three-dimensional 
topographic data 210 of the geographic information system. This 
map processing means 206 is mainly constituted of a photographic 
area computing section (image frame computing 212) that obtains 

10 a photographic area on the map of the geographic information 
system corresponding to a photographic area of a shot image, 
which the photographic means has taken; an image transformation 
section (image transformation . 213) that transforms the 
mentioned shot image in conformity with a photographic area 

15 having been obtained by the image frame computing 212; and a 
■ monitor (e.g., super impose 214) that displays the mentioned 
transformed shot image super imposed on the mentioned 
photographic area of the mentioned map. 

At the map processing 206, first, image frame computation 

20 is executed in image frame 212 in which a shooting position 
in the air is specified three-dimensionally with information 
signals regarding an airframe position, and a photographic area 
(=photographic frame) of the ground surface having been shot 
is obtained by computation based on posture of the camera and 

25 airframe with respect to the ground surface. Image 
transformation 213 is performed in conformity with this image 
frame. This image transformation is to transform the image 
so that an image becomes, e.g., a shape close to trapezoid, 
or rhombus in which shape the image conforms to the map. Then, 

30 the transformed image is superposed (pasted) in superposition 
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step 214 onto a map of the geographic information system. 
Thereafter, this resultant picture is displayed with monitor 
display means 211 such as CRT. 

Fig . 3 is a photograph in which a shot image 302 is superposed 
5 on a map 301 of the geographic information system with a 
photographic frame 303 corresponding to the map. Numeral 304 
designates a flight path of the airframe, and numeral 305 
designates an airframe position (camera position) . The map 
processing including the above-described transformation 

10 processing with the map processing means 206 causes an image 
to be in coincidence with the map substantially at all points, 
as shown in Fig. 3, and makes it easy to ascertain conformity 
between image information andmap, thereby enabling to determine 
a target point easily. 

15 Further, as shown in Fig. 3, an image of the image frame 

having been shot with the camera, can be displayed being 
superposed on the map, as well as it can be done easily to erase 
the shot image 302 and display only the image frame 303 . Herein 
the shot image 302 is superposed on the two-dimensional map. 

20 Accordingly, for example, a place of the disaster occurrence 
(e.g., building on fire) is visually confirmed with the shot 
image 302, and the position thereof is checked (clicked) on 
the shot image 302. Thereafter, the image 302 is erased, and 
the two-dimensional map under the shot image 302 is displayed 

25 leaving only the image frame 303 displayed, thus enabling to 
rapidly recognize a place on the map of the position having 
been checked on the shot image. Further, supposing that 
displayed images on a monitor are arranged to display in a 
definite direction regardless of a direction of the camera, 

30 the determination of a target point becomes still easier. 



Embodiment 2 . 

According to this second embodiment, a current position 
of the airframe 101 is measured, a photographic frame of the 
5 ground having been shot from on board is computed, and an image 
having been shot is transformed and pasted onto a map of the 
geographic information system in conformity with the 
photographic frame. At the time of executing a comparison 
between a shot image and a map is done, plural pieces of shot 

10 images are sampled in succession in cycles of a predetermined 
time period from images having been continuously shot. Then 
a series of plural images are pasted onto the map of the geographic 
information system to be displayed, and a target point of land 
is specified from the images pasted onto the map. 

15 Fig. 4 shows a monitor display screen according to this 

method. Numeral 304 designates a flight path of the airframe. 
Numeral 305 designates an airframe position (camera position) . 
Images having been shot with the camera along the flight path 
304 are sampled with a predetermined timing to obtain each image 

20 frame, and the shot images are transformed and processed so 
as to conform to the image frames and pasted onto the map 301. 
Numerals 302a to 302f are pasted images. Numerals 303a to 303f 
are image frames thereof. 

The computation of a photographic frame and the 

25 transformation of an image into each image frame are executed 
by computing with the use of camera information and posture 
information of the airframe at the time of. taking a shot as 
.described in the first embodiment. It is preferable that a 
sampling period for each image is changed in accordance with 

30 a speed of. the airframe. Normally, a sampling period is set 
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to be shorter when the airframe flies at high speed, and the 
sampling period is set to be longer when the airframe flies 
at low speed. 

According to this second embodiment, it becomes possible 
5 to identify situations on the ground while confirming the 
situations of a wide range of ground surface with a map and 
plural pieces of continuous images, thereby enabling to 
determine a target point of land more effectively. 

10 Embodiment 3. 

According to this third embodiment, a current position of 
the airframe 101 and a rotation angle and inclination (pan and 
tilt=posture of the camera) of the camera 102. with respect to 
the airframe are measured, and a photographic frame of the ground 

15 having been shot from on board is computed based on this camera- 
posture. Then the image having been shot are transformed and 
pasted onto a map of the geographic information system in 
conformity with this photographic frame, and the comparison 
between the shot image and map is executed. 

20 According to this third embodiment, a photographic frame 

is computed based on posture of the camera acting as photographic 
means, thereby enabling to identify situations of the ground 
with higher precision while confirming a positional relation 
between the shot image and the map. 

25 Now, relations between the airframe 101 and the camera 102 

are shown in Figs. 5. On the assumption that the camera 102 
is housed in the gimbal 112, and the airframe 101 flies level,, 
as shown in Figs. 5 (b) and (c) , an inclination of the camera 
102 is outputted as an inclination of the airframe 101 with 

30 respect to a central axis (-tilt), and a rotation angle (pan) 
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of the camera 102 is outputted as a rotation angle from a traveling 
direction of the airframe 101. That is, in the state of (b) , 
the camera 102 faces right below so that an inclination is 0 
degree. In the state of (c) , an inclination 6 of the camera 
5 102 is shown to be an inclination with respect to the vertical 
plane. 

The method of computing a photographic frame of the camera 
can be obtained with rotational movement and projection 
processing of a rectangle (image frame) in 3D coordinates as 

10 a basis of computer graphics. Basically, a photographic frame 
of the camera is processed by transformation between camera 
information and airframe information, and a graphic frame in 
the case of projecting this photographic frame to the ground 
is computed, thereby enabling to obtain an intended image frame . 

15 A method of computing each coordinate in 3D coordinates is 
obtained by using the following matrix calculation method. ■ 

1) Computing a photographic frame in the reference state 

First, as shown in Fig. 6 (a), positions of four points 
20 of an image frame are computed as relative coordinates, letting 
a position of the airframe an origin. The photographic frame 
is computed into a reference position based on a focal length, 
angle of view and altitude of the camera thereby obtaining 
coordinates of four points. 

25 

2) Computing positions of four points after the rotation about 
a tilt of the camera (Z-axis) 

As shown in Fig. 6 (b) , a photographic frame is rotated, 
about Z-axis in accordance with a tilt angle 6 of the camera. 
30 Coordinates after rotation are obtained by transformation with 
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the following expression 1. 



[Expression 1] 



[*' y z' l]=[x y z l] 



' cos9 sinO 0 0 
-sin 6? cosO 0 0 



0 



0 



1 0 



0 



0 



0 1 



3) Computing positions of four points after the rotation about 
an azimuth of the camera (y-axis) 

As shown in Fig. 6 (c) , a photographic frame is rotated 
about y-axis in accordance with an azimuth 6 of the camera. 
Coordinates after the rotation are obtained by transformation 
with the following expression 2. 
[Expression 2] 



4) Calculating a graphic frame of projecting the image frame 
after rotational processing from an origin (airframe position) 
to the ground surface (y-axis altitude point) 

As shown in Fig. 6 (d) , a projection plane (photographic 
frame) is obtained by projecting the photographic frame to the 
ground surface (y-axis altitude) . Coordinates after 

projection are obtained by transformation with the following 
expression 3 . 
[Expression 3] 



[x> y z x 



y z 1] 



cos6 0 -sin6* 0 

0 10 0 

sin£? 0 cos6? 0 

0 0 0 1 



[x' y z' l]-[x y z l] 



10 0 0 

0 10 lid 
0 0 1 0 
0 0 0 0 
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Generalized homogenous coordinate system [X, Y, Z, W] is 
obtained with the following expression 4. In addition, d is. 
a sea level altitude. 
[Expression 4] 



[X Y Z W]=[x y z y/d] 



Next, the expression 4 is divided by W (=y/d) and returned 
to be in 3D, resulting in the following expression 5. 
[Expression 5] 



X_ Y_ Z_ " 
WWW 



= [xp yp zp l]. 



y/d 



y/d 



Embodiment 4 . 

According to this fourth embodiment, a current position 
of the airframe 101 and an elevation angle and roll angle of 

15 the airframe 101 are measured, and a photographic frame of the 
ground having been shot from on board is computed based on the 
elevation angle and roll angle. Then an image having been shot 
is transformed and pasted onto a map of the geographic 
information system in conformity with the photographic frame 

20 thereof, and the comparison between the shot image and the map 
is executed. According to this fourth embodiment, a 
photographic frame is computed based on the positional 
information of the airframe 101 with respect to the ground, 
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thereby enabling to identify situations of the ground with higher 
precision while confirming a positional relation between the 
shot image and map. 

Now, as to relation between the airframe and the camera, 
5 let it be assumed that the camera 102 is fixed to the airframe 
101 (that is, the gimbal is not used) as shown in Fig. 7. In 
the case where the airframe 101 itself flies horizontally to 
the ground as shown in Fig. 7 (b) , the camera 102 faces right 
below so that inclination of the camera 102 becomes 0 degree.- 
10 In the case where the airframe 101 is inclined as shown in Fig. 
7 (c) , this inclination gives a posture of the camera 102 and, 
therefore, a photographic frame of the camera is computed based 
on an elevation angle (pitch) and roll angle of the airframe 
101. 

15 

1) Computing a photographic frame in the reference state 

As shown in Fig. 8 (a) , positions of four points of an image 
frame are computed as relative coordinates, letting a position 
of the airframe an origin. The photographic frame is computed 
20 into a reference position based on a focal length, angle of 
view, and altitude of the camera, thereby obtaining coordinates, 
of four points. 

2) Computing positions of four points after the rotation about 
25 a roll of the airframe (x-axis) 

As shown in Fig. 8 (b) , the photographic frame is rotated 
about x-axis in. accordance with a roll angle 6 of the airframe 
with the following expression. Coordinates after rotation are 
obtained by transformation with the following expression 6. 

30 
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[Expression 



6] 



be 



!]-[: 



x y z 



1] 



"10 0 0 

0 cos0 sin0 0 

0 -sinO cos0 0 

0 0 0 1 



3) Computing positions of four points after the rotation about 
a pitch of the airframe (z-axis) 

As shown in Fig. 8 (c) , the photographic frame is rotated 
about the z-axis in accordance with a pitch angle 6 of the 
airframe. Coordinates after rotation are obtained by 
transformation with the following expression 7. 



10 
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[Expression 7] 



x y z 



cos0 sin£ 0 0 

-smO cosd 0 0 

0 0 10 

0 0 0 1 



20 



4) Calculating a graphic frame of projecting the image frame 
after rotation processing from an origin (airframe position) 
to a ground surface (y-axis altitude point) 

As shown in Fig. 8 (d) , a projection plane (photographic 
frame) is obtained by projecting the photographic frame to the- 
ground surface (y-axis altitude) . Coordinates after 
projection are obtained by transformation with the following 
expression 8. 
[Expression 8] 
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[x* y' z' l] = [x y z l] 



10 0 0 

0 1 0 ltd 

0 0 1 0 

0 0 0 0 



Generalized homogenous coordinate system [X, Y, Z, W] is 
obtained with the following expression 9. 
[Expression 9] 

[X Y Z w]-[x y z y/d] 

Next, the expression 9 is divided by W f (=y/d) and returned 
to be in 3D resulting in the following expression 10. 
[Expression 10] 



www 



[xp yp zp l]. 



y/d 



y/d 



Embodiment 5 . 

In this fifth embodiment, a current position of the airframe 
101, a rotation angle and inclination of the camera 102 with 
respect to the airframe, and further an elevation angle and 
roll angle of the airframe 101 are measured, and a photographic 
frame of the ground having been shot from on board is computed 
based on the information. Then an image having been shot is 
transformed and pasted onto a map of the geographic information 
system in conformity with the photographic frame thereof, and 
the comparison between the image and the map is executed. 
According to this fifth embodiment, a photographic frame is 
computed based on posture information of the camera and posture 
information of the airframe, thereby enabling to identify 
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situations of the ground with higher precision while confirming 
a positional relation between the image and map. 

Now, as to relation between the airframe 101 and the camera 
102, supposing that the camera 102 is housed in the gimbal 112 
5 as well as the airframe 101 flies in any posture as shown in 
Fig. 9, an inclination and rotation angle of the camera 102 
are outputted from the gimbal 112 as shown in Fig. 8(b). 
Furthermore, an elevation angle and roll angle of the airframe 
101 itself with respect to the ground are outputted from the 
10 gyro. 

The method of computing a photographic frame of the camera 
can be obtained with rotational movement and projection 
processing of a rectangle (image frame) in 3D coordinates as 
a basis of computer graphics. Basically, a photographic frame 

15 of the camera are processed by transformation with camera 
information and airframe information, and a graphic frame in 
the case of projecting this photographic frame to the ground 
is computed, thereby enabling to obtain an intended image frame . 
The method of calculating each coordinate in 3D coordinates 

20 is obtained by using the following matrix calculation method. 

1) Computing a photographic frame in the reference state 

As shown in Fig. 10 (a) , positions of four points of an 
image frame are computed as relative coordinates, letting a 
25 position of the airframe an origin. A photographic frame is 
computed into a reference position based on a focal length,, 
angle of view, and altitude of the camera thereby obtaining 
coordinates of four points. 

30 2) Computing positions of four points after the rotation about 
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a tilt of the camera (Z-axis) 

As shown in Fig. 10 (b) , transformation of rotating a shot 
image about Z-axis in accordance with a tilt angle 6 of the 
camera is executed. Coordinates after rotation are obtained 
by transformation with the following expression 11. 
[Expression 11] 



y z s l]=[x y z l] 



cos0 sin0 0 0 

-sin 6 cosO 0 0 

0 0 10 

0 0 0 1 



3) Computing positions of four points after the rotation about 
an azimuth of the camera (y-axis ) 

As shown in Fig. 10 (c) , transformation of rotating a 
photographic frame about y-axis in accordance with an azimuth 
9 of the camera is executed. Coordinates after rotation are' 
obtained by transformation with the following expression 12. 
[Expression 12] 



f z' l]-[x y z l] 



cos0 0 -sin 0 0 

0 10 0 

sin 0 0= cos0 0 

0 0 0 1 



4) Computing positions of four points after the rotation about 
a roll of the airframe (x-axis) 

As shown in Fig. 10 (d) , transformation of rotating a 
photographic frame about x-axis in accordance with a roll angle 
8 of the airframe is executed. Coordinates after rotation are. 
obtained by transformation with the following expression 13. 
[Expression 13] 
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[x* y z x l]-[x y z l] 



'10 0 0 

0 cos£ sind 0 

0 -sin 6 cos£? 0 

0 0 0 1 



5) Computing positions of four points after the rotation about 
a pitch of the airframe (z-axis) 

As shown in Fig. 10 (e) , transformation of rotating a 
photographic frame about z-axis in accordance with a pitch angle 
8 of the airframe is executed. Coordinates after rotation are 
obtained by transformation with the following expression 14. 
[Expression 14] 

cos@ sind 0 0" 
-sinO cosO 0 0 
0 0 10 
0 0 0 1 



[x* y z r l]=[x y z l] 



6) Calculating a graphic frame of projecting the image frame- 
after rotational processing from an origin (airframe position) 
to a ground surface (y-axis altitude point) 

As shown in Fig. 10 (f ) , a projection plane (photographic 
frame) is obtained by projecting the photographic frame to the 
ground surface (y-axis altitude) . Coordinates after 
projection are obtained by tr ansformation with the f ollowinci 
expression 15. 
[Expression 15] 

[x' y z' !]=[* y z 1] 



10 0 0 

0 1 0 lid 

0 0 1 0 

0 0 0 0 



7) Generalized homogenous coordinate system [X, Y, Z, W] is 
obtained with the following expression 16. 

[Expression 16] 

[X Y Z w] = [x y z y/d] 

8) Next, the expression 16 is divided by W 1 (=y/d) and returned 
to be in 3D resulting in the following expression 17. 
[Expression 17] 

I F Z 

w w w 

Embodiment 6. 

In this sixth embodiment, a current position of the airframe 
101, a rotation angle and inclination of the camera 102 with 
respect to the airframe, and further an elevation angle and 
roll angle of the airframe 101 are measured, and a photographic 
frame of the ground having been shot from on board is computed 
into a map of the geographic information system. In computing 
processing of four points of this photographic frame, 
topographic altitude data is utilized, and a flight position 
of the airframe 101 is compensated to compute the photographic 
frame. Then an image having been shot is transformed in 
conformity with the photographic frame thereof and pasted onto 
a map of the geographic information system, and the comparison 
between the shot image and map is executed. 

. According to this sixth embodiment, the compensation is 
executed with altitude topographic information of the surface 
ground using information about a position and altitude of the 
airframe, airframe posture information and posture information 



[xp yp zp l]< 



X d -5- 1 



y/d y/d 



22 



of the camera, and a photographic frame is computed, thereby 
enabling to identify with higher precision situations of the 
ground while confirming a positional relation between the image 
and the map. 

In the foregoing fifth embodiment, a sea level altitude 
obtained from the GPS apparatus is employed as an altitude of 
the airframe in computing processing of a photographic frame 
onto the ground surface after rotation: whereas, in this sixth 
embodiment, as shown in Fig. 11, a ground surface altitude 
(relative altitude d = sea level altitude - ground surface 
altitude) at a shooting point is employed as an altitude of 
the airframe utilizing a topographic altitude information of 
the ground surface. In this manner, computing four points of 
a photographic frame is executed. 

1) Calculating a graphic frame of projecting an image frame 
after rotational processing from an origin (airframe position) 
to the ground surface (y-axis altitude point) 

A projection plane is obtained by projecting the 
photographic frame to the ground surface (y-axis altitude) . 
Coordinates after projection are obtained by transformation 
with the following expression 18. 
[Expression 18] 

[x' y z x y z l] 

Generalized homogenous coordinate system [X, Y, Z, W] is 
obtained with the following expression 19. 



10 0 0 
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0 0 0 0 



[Expression 19] 

[X Y Z W]=[x y z y/d] 

Next, the expression 19 is divided by W 1 (=y/d) and restored 
to be in 3D resulting in the following expression 20. 
5 [Expression 20] 

I K Z 
WWW 

A relative altitude d, which is used herein, is obtained 
by subtracting a topographic altitude at a target point of land 
from an absolute altitude from the horizon, which is obtained 
10 from the GPS apparatus. Further this relative altitude from 
the camera is utilized, thereby enabling to compute with higher 
precision the position of a photographic frame. 

Embodiment 7 . 

15 In this seventh embodiment, at the time of measuring a 

current position of the airframe 101, computing a photographic 
frame of the ground having been shot from on board on a map 
of the geographic information system, transforming an image 
having been shot in conformity with the photographic frame 

20 thereof and pasting it, and executing the comparison between 
the shot image and map, plural pieces of shot images to be pasted 
in succession on the map are displayed being pasted continuously 
onto the map of the geographic information system. Then a target 
point of land is specified with the pasted images on the map. 

25 In the processing of pasting plural pieces of shot images 

onto a map of the geographic information system, the layout 
is performed in accordance with the computed photographic frames , 
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a joint state of overlap part of each shot image is confirmed, 
and the images are moved so that overlap condition of the images 
may be of the largest extent to execute the positional 
compensation. Then, the shot images are transformed in 
conformity with the photographic frames on the map of the 
geographic information system with the use of the compensation 
values, and paste processing is performed. 

Procedures thereof are shown in Fig. 12. For example, two 
pieces of shot images 1 (A) and 2 (B) , which are taken as the 
airframe 101 travels, are overlapped, and an overlap part 
(internal part of a solid frame C of the drawing) is detected. 
Then the images A and B are moved relatively so that the overlap 
condition of the images may be of the largest extent, a positional 
compensation value at the time of joining is obtained, the 
positional compensation . D is executed, and the images A and 
B are joined. The positional compensation is carried out in 
image joining * compensation 215 of Fig. 2. 

According to this seventh embodiment, plural pieces of 
continuous images provide a more precise joining, thereby 
enabling to identify situations of the ground while confirming 
situations of a wider range of ground surface. 

Embodiment 8 . 

In this eighth embodiment, a current position of the airframe 
101, a mounting angle and inclination of the camera 102 with 
respect to the airframe, and further an elevation angle and 
roll angle of the airframe 101 are measured. Then a photographic 
frame of the ground having been shot from on board is computed, 
the image is transformed in conformity with the photographic 
frame thereof to be pasted onto a map of the geographic 



information system, and the comparison between the sot image 
and map is executed. 

In the case of executing this processing, it comes to be 
important that various information, which are transmitted from 
the on-board system 100, are received at the ground system 200 
fully in synchronization. To achieve this synchronization, 
it is necessary to adjust a processing time period such as 
processing time period of flight position measurement means, 
processing time period of posture measurement means with the 
gimbal of camera and a processing time period of image 
transmission, and to transmit them in synchronization with the 
shot image. To actualize this synchronization, a buffer is 
provided in the construction of Fig. 1, and image signals of 
the camera on board are temporarily stored with storage means 
113 in this buffer and transmitted to the ground system 200 
in synchronization with the forgoing information after the delay 
of a computing time period for airframe positional detection 
by, e.g. , GPS. 

This relation is described with reference to Fig. 13. A 
time period T is required for the. airframe 101 to receive a 
GPS signal and detect an airframe position, and during this 
time period the airframe 101 travels from a position PI to a 
position P2 . Therefore at the instant of completing a 
positional detection of the airframe, a region, which the camera 
102 shoots, comes to be a region apart from the region, which 
the camera 102 has shot at the position PI, by a distance R 
resulting in occurrence of error. 

Fig. 13 (b) is a time-chart showing procedures of correcting 
this error. An image signal is temporarily stored in the buffer 
during a GPS computing time period T from a GPS observation 



10 



point tl for airframe positional detection, and the image signal 
having been stored temporarily is transmitted together with 
airframe position, airframe posture, camera information and 
the like at the instant of t2 . 

According to this eighth embodiment, a photographic frame 
is computed based on mounting information of the photographic 
device, thereby enabling to identify with higher precision 
situations of the ground while confirming a positional relation 
between the image and map. 

Further, according to each of the foregoing embodiments, 
an image frame is computed, thereafter a shot image is 
transformed in conformity with this image frame, and this 
transformed image is superposed and pasted onto a map . However, 
it is preferable that a photographic area on the map 
15 . corresponding to a shot image, which photographic means has 
taken,' is merely obtained, and the shot image is superposed 
on this area of the map to be displayed. 

Furthermore, according to each of the foregoing embodiments , 
map processing is executed at the ground system based on 
20 information tobe transmitted from the on-board system. However, 
this map processing is not limited thereto, and it is preferable 
that the on-board system is provided with a monitor such as 
display, the map processing is executed at the on-board system, 
and the processed map is displayed on the monitor of the on-board 
system; or that information having been processed is transmitted 
to the ground system, and displayed at the ground system. 

Embodiment 9 . 



25 



30 



According to this ninth embodiment, so-called land marks, 
for example, a cross point or station or a large building corner 
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that show remarkable points on the map are extracted from a 
shot image; and the corresponding landmark is extracted from 
a region corresponding to the photographic area on the map. 
Further, parameters for image frame computing (hereinafter, 
5 showing information of airframe position, airframe posture and 
camera posture, and camera set information for use in computing 
a photographic frame, being a photographic area of the camera 
on the ground surface) are adjusted so that the landmarks of 
the image and the map may be in coincidence, whereby the image 

10 is transformed and displayed being superposed on a GIS screen 
with high precision. 

Hereinafter, descriptions are made referring to the 
drawings. Fig. 14 is a block diagram showing the ninth- 
embodiment. Additionally, in Fig. 14, diagrammatic 

15 representations of the antenna 103, multiplex modulator 109, 
signal conversion means 110, tracking means 111, temporary 
storage means 113, transmission means 104 , receiving means 201, 
tracking means 202, signal conversion means 2 03 , and multiplex 
demodulator 204 are omitted. Fig. 15 is a function explanatory 

20 diagram for explaining map processing means. 

With reference to Fig. 14, current positional information 
is obtained with airframe position measurement means 108 such 
as GPS apparatus that is mounted on a flight vehicle (=airframe) 
such as helicopter, and the airframe positional measurement 

25 is performed. Furthermore, the airframe 101 comprises, e.g., : 
gyro, and posture, i.e., an elevation angle (=pitch) and roll 
angle are measured with this airframe posture measurement means 
107. Photographic means 105, being the camera 102 mounted on 
the airframe 101 takes a shot of the ground, and outputs image 

30 signals thereof as well as outputs camera information such as 



zoom of the camera . The camera 102 is attached to, e.g., gimbal, 
and a rotation angle (=pan) and inclination (=tilt) of the camera 
is measured with this camera posture measurement means 106.- 
Outputs from these airframe position measurement means 108 , 
airframe posture measurement means 107, photographic means 105, 
and camera posture measurement means 106 are inputted to signal 
processing means 205 and signal-processed respectively. Image 
signals of camera shooting are converted to a moving image data 
207 and a still image data 208. Outputs from the signal 
processing means 205 and a two-dimensional map data 209 are 
inputted to map processing means 226, and the map processing 
is executed. 

The map processing means 226 includes functions shown in 
Fig. 15. In the map processing means 226, as shown in Fig. 
15, the processing is executed based on a moving image data 
207 and a still image data 208, being image signals, and 
information signals of airframe position, airframe posture, 
and camera posture, and a two-dimensional map data 209 of the 
geographic information system. 

In the map processing means 226, first image frame computing 
212 is executed, in which a shooting position in the air is 
specified three-dimensionally, and a photographic area 
(=photographic frame) of the ground surface having been shot 
is obtained by computation based on posture of the camera and 
airframe with respect to the ground surface. Then, landmark 
extraction 220 is executed to an extent corresponding to the . 
photographic area and its vicinity on a map of the geographic 
information system, and landmark extraction 221 is executed 
also from a still image data 208. Landmark comparison 222 for 
causing these landmarks in coincidence is executed. Image 



transformation • compensation 223 is executed based on a result 
of the landmark comparison 222, and a superposed display posit ion 
of a shot image onto the map is compensated. Thereafter, 
superposition 214 of the image on the map of the geographic 
information system is executed. Finally, this superposed' 
picture is displayed on a monitor with monitor display means 
211 such as CRT. 

Now, operations are described based on a flowchart of Fig. 
16. First, an airframe position, being an output from airframe 
position measurement means 108, a pitch elevation angle and 
roll angle, being an output from airframe posture measurement 
means 107, a pan and tilt, being an output from camera posture 
measurement means 106, a zoom of the camera 102, being an output 
from photographic means 105, a still image data 208 obtained 
with signal processing means 205, and a two-dimensional map 
data 209 are read in as input data respectively (S21) . Next, 
the image frame computing 212 is executed using an airframe- 
position, pitch elevation angle, roll angle, and a pan, tilt 
and zoom of the camera as parameters (S22) . Subsequently, the 
landmark extraction on the map of a geographic information system 
is executed about a region corresponding to a photographic frame 
obtained by the image frame computing 212 (S23) . In the case 
where any landmark is extracted in S23, the corresponding 
landmark is extracted from a still image data 208 (S24) (S25) . 

In the case where the landmark is extracted also from an 
image in S25, the corresponding landmarks that are obtained 
in S23 and S25 are compared with each other, and parameter (for 
example, pan • tilt) values having been used in the image frame 
computing of S22 are compensated so that these landmarks are 
in coincidence (S26) (S27) (S28). Further, the photographic 
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frame is computed again based on the compensation value of 
parameters having been obtained in S28, and a still image data 
208 is transformed in conformity with this photographic frame 
and displayed being superposed on a map of the geographic 
5 information system (S29) (S30)(S31). 

In the case where any landmark is not extracted in S23 or 
S25, a still image data 208 is transformed in conformity with" 
a photographic frame obtained in S22, and displayed being 
superposed on a map of the geographic information system 
10 (S24) (S26) (S30) (S31) . Fig. 17 shows a pitch elevation angle, 
rotation angle, and a pan and tile of the camera, being angle 
parameters for use in the image frame computation 212. 

For the computing method of a photographic frame, the 
above-described method is employed. A photographic frame in 
15 the reference state is rotationally- processed with each angle 
parameter, and thereafter projected onto the ground surface, 
whereby a photographic area of the camera on the ground surface, 
that is, a photographic frame is obtained. As shown in Figs. 
18, when x-axis is laid in airframe traveling direction, z-axis* 
20 is laid in vertically upward direction with respect to the ground 
surface, and y-axis is laid so as to be vertical to these x-axis 
and z-axis, letting an airframe position an origin, the specific 
computation is as follows: 

Computing a photographic frame in the reference state 
25 Rotation about y-axis based on a tilt of the camera 
Rotation about z-axis based on a pan of the cameral 
Rotation about x-axis based on a roll angle of the airframe 
Rotation about y-axis based on a pitch elevation angle of the 
airframe 

30 Projection onto the ground surface (horizontal surface of 
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absolute altitude (=sea level altitude) 0) 

Fig. 18(a) shows the state in which a photographic frame 
42 is computed into the reference state. Fig. 18(b) shows the 
state in which the photographic frame 42 of the reference state 
5 is rotationally processed with each angle parameter, and 
thereafter projected onto the ground surface. 

The method of compensating a pan and tilt of the camera 
is now described referring to Fig. 19 . When letting an airframe 
altitude h, a measured value of tilt 6 r a measured value of 
10 pan </> , landmark coordinates on an image (x, y) and landmark 
coordinates on the map (x 0 , yo) t values of tilt and pan after 
the compensation 6 0 , $ o can be obtained by working out the 
following expression 21. 
[Expression 21] 

(h -tan^o -cos^ 0 = x 0 
|A-tan0 o -sinA = y 0 

where : landmark coordinates (x 0 , yo) on the map to compare herein 
are coordinates after the following transformation. 
Counter-rotation about y-axis based on a pitch elevation angle 
of the airframe 

20 Counter-rotation about x-axis based on a roll angle of the 
airframe 

Projection onto the ground surface (horizontal surface of 
absolute altitude (=sea level altitude) 0) 

Fig. 20(a) is a picture of a photographic frame 42 and a 
25 shot image 43 being superposed onto a map 41 of the geographic 
information system without compensation according to the 
invention. Fig. 20(b) is a picture after being subjected to 
the compensation according to the invention, showing a 
photographic fame 42 and a shot image 43 being superposed on 
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the map 41 of a geographic information system. Numeral 4 4 
indicates an airframe position (camera position) . By the 
processing with map processing means 226 including the 
above-described compensation processing, an image and a map 
are in coincidence at all points, as shown in Fig. 20(b) , thus 
enabling to carry out a superposed display with high precision, 
and to understand situations of the ground surface having been 
shot more easily and rapidly. 

According to this ninth embodiment, not only it is possible 
to correct measuring error of various measurement devices that 
measure each parameter; but also it becomes possible to correct 
error having occurred due to lag in timing between shooting 
and data-getting of camera posture information (pan • tilt) in 
the case of superposing and displaying an image having been 
shot during operation of a camera that is mounted on the airframe 
on the map. 

Embodiment 10. 

This tenth embodiment is a method of making the parameter 
adjustment of the above-mentioned ninth embodiment not by the 
compensation of pan and tilt but by the compensation of posture 
information (roll and pitch) of the airframe, thereby 
compensating position of a photographic frame. The 
compensation of roll and pitch is executed with the following 
computation. 

When letting landmark coordinates on an image at the time 
of completing the rotational processing with a tilt and pan 
(xi, yi, Zi) , landmark coordinates (x 2 , y2, Z2) at the time of 
having executed the rotational processing with a roll 6 and 
pitch $ is obtained with the following expression 22. 



[Expression 22] 

cos<9 0 sinflYl 0 
(x 2 y 2 z 2 ) = (x l y, zA 0 1 0 



- sin 0 0 cos 9 



0 cos (f> sin <fi 
0 - sin <b cos 6 



Further, when performing the projection onto the ground surface, 
landmark coordinates (x, y, z) are obtained with the following 
5 expression 23. 
[Expression 23] 

(x y z)=(x 2 y 2 z 2 )- — 

Herein, an alphabet h is an airframe altitude, and 6 , 4> 
satisfying the following expression 24 
10 • [Expression 24]. 

(x(0,0) = X Q 

when letting landmark coordinates on the map (x 0 , yo) , are roll 
0 0/ pitch 0o after the compensation. 

According to this tenth embodiment, since the camera is 
15 fixedly attached to the airframe, and mounted so that an angle 
of pan and tilt is not varied, the compensation of parameters 
in the more real state can be made by correcting posture 
information of the airframe that is a roll and pitch even in 
the case where the compensation with a pan an tilt is ineffective , 
20 thus enabling to carry out a more precisely superposed display. 
As a result, it is possible to understand situations of the 
ground surface having been shot more easily and rapidly. 

Embodiment 1 1 . 

25 According to this eleventh embodiment , 2 points of landmarks 

are extracted, and the altitude compensation of the airframe 
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is made with a distance between these 2 points. In the case 
where 2 points of landmarks are extracted in S23 of the ninth 
embodiment (Fig. 16), the corresponding 2 points of landmarks 
are likewise extracted also from a still image data (S2 4 ) (S25) . 
5 In the case where the corresponding landmarks are extracted 

also from an image in S25, the landmarks having been obtained 
in S23 and S25 are compared, and an airframe altitude is 
compensated so that a distance between 2 points of landmarks 
on the image and a distance between 2 points of landmarks on 

10 theGISmap (in this case, since an airframe altitude is obtained 
as an absolute altitude fromthe sea level with the GPS apparatus, 
a relative altitude from the ground surface will be obtained 
by this altitude compensation) (S27) (S28) . 

Further, a photographic frame is computed again based on 

15 the compensation values of parameters that are obtained in S28, 
a still image data 208 is transformed in conformity with this 
photographic frame and displayed being superposed on a map of 
a geographic information system (S29) (S30) (S31) . 

As seen from Fig. 21(b), an altitude (relative altitude) 

20 h' having been compensated with a distance between landmarks 
according to the invention is obtained with the expression of 
(relative altitude )=( absolute altitude) X (distance between 
2 points of landmarks on a map) / (distance between 2 points of 
landmarks on an image) , 

25 letting an absolute altitude of the airframe h. In the drawing, 
E is a distance on the map and F is a distance on the image. 

By the processing with map processing means 226 including 
the above-described compensation processing, a shot image with 
respect to a point of land of which ground surface is higher 

30 than the sea level can be displayed being superposed with high 
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precision, thereby enabling to understand situations of the 
ground surface having been shot more easily and rapidly. 

Embodiment 12. 

This twelfth embodiment is intended to make it possible 
that a shot image and map are displayed being superposed with 
higher precision by compensating parameters in accordance with 
the number of landmarks . In the case where 2 points of landmarks 
are extracted in S22 of the foregoing ninth embodiment (Fig. 
16) , the corresponding 2 points of landmarks are likewise 
extracted also from a still image data 208 (S24) (S25) . In the 
case where the landmarks are extracted also from an image in 
S25, the corresponding landmarks obtained in S23 and S25 are 
compared. 

First, parameter (pan and tilt) values having been used 
in the image frame computing of S22.are compensated so that 
the first corresponding landmarks are in -coincidence, and next 
airframe posture parameter (roll and pitch) values are 
compensated so that a difference between the second 
corresponding landmarks are corrected (S27) (S28) . Further, 
a photographic frame is computed again based on. the compensation 
values of each parameter obtained in S28, and a still image 
data 208 is transformed in conformity with this photographic 
frame and displayed being superposed on the map of the geographic 
information system (S29) (S30) (S31) . 

Fig. 22 is a diagram explaining this compensation, and in' 
which black circle marks indicate landmarks on the map and filled 
triangle marks indicate landmarks on the image. Fig. 22(a) 
shows the state in which a shot image is displayed being 
superposed on the GIS map; Fig. 22(b) shows the state after 
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the altitude compensation according to the foregoing eleventh 
embodiment has been executed; Fig. 22(c) shows the state after 
the pan and tilt compensation has been executed thereafter; 
and Fig. 22(d) shows the state after the roll and pitch 
compensation has been further executed. 

According to this twelfth embodiment, even in the case where 
it is difficult that a shot image and a map are displayed being 
superposed with high precision over the entire photographic 
area by the adjustment of parameters with only 1 point of 
landmarks in coincidence, the superposed display with higher 
precision can be achieved by using 2 points of landmarks , thereby 
enabling to understand situations of the- ground surface having 
been shot more easily and rapidly. 

Embodiment 13. 

According to this thirteenth embodiment, in the case where' 
not less than 3 points of landmarks are extracted, parameter 
compensation values between all the two points, and an average 
value thereof is used as a parameter compensation value. In 
the case where a plurality of landmarks of not less than 2 points 
are extracted in S23 of the ninth embodiment (Fig. 16), the 
corresponding plural landmarks of not less than 2 points are 
likewise extracted also from a still image data 208 (S24) (S25) . 

In the case where landmarks are extracted also from an image 
in S25, the corresponding 2 points are picked up from the 
landmarks having been obtained in S23 and S25, and respective 
comparisons are executed, thereby obtaining a compensation 
value of parameters. This processing is executed as to all- 
selections of 2 points of landmarks, whereby a plurality of 
parameter compensation values are obtained, an average of these 



compensation values as to respective parameters is obtained, 
and these average values are used as a compensation value of 
respective parameters (S27)(S28). Further, a photographic 
frame is computed again based on the compensation value of 
parameters having been obtained in S28, and a still image data 
208 is transformed in conformity with this photographic frame 
and displayed being superposed on a map of the geographic 
information system (S29) (S30) (S31) . 

By the map processing including the mentioned compensation 
processing, as compared with the case of compensating the 
superposed display of an image and map based on positions of 
1 or 2 points of landmarks, it is now possible to achieve the 
superposed display with higher precision, thereby enabling to 
understand situation of the ground surface having been shot 
more easily and rapidly. 

Embodiment 14. 

This fourteenth embodiment relates to superposed display 
processing onto the map in the case where plural pieces of images 
20 are shot continuously in cycles of a predetermined time period 
and a series of plural images are provided as a still image 
data. The extraction processing of landmarks is carried out 
with respect to an obtained still image . As a result, supposing 
that landmarks are extracted, the compensation is executed by 
25 the comparison with the GIS map. However, landmarks cannot 
always be extracted from all the still images. In the live 
display processing of performing the superposed display while 
taking a shot, it is difficult to " instantly execute image 
processing to extract landmarks and perform the compensation 
30 with respect to all shot images due to processing time period. 
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Therefore, as for the superposed display on the map of a 
still image in which landmarks are not extracted, image fame 
computation is executed again based on compensation values at 
the time of the last compensation, an image is transformed in 
5 conformity with the photographic frame having been obtained 
and displayed being superposed on a map of the geographic 
information system, thereby achieving improvement in precision 
of the superposed position. 

This processing corresponds to S24, S26, S32, S33, S31 of 
10 Fig. 23. In the case where any corresponding landmark is 
extracted in S24, the same processing as in the ninth embodiment 
is executed. Fig. 24 shows a monitor display screen according 
to this method. Numeral 41 designates a map; numeral 44 
designates an airframe position (camera position) ; and numeral 
15 45 designates a flight path of the airframe. Images having 
been shot with the camera along the flight path 45 are sampled 
with a predetermined timing, subjected to the superposed 
positional compensation respectively, and thereafter displayed 
being superposed on a map 41 of the geographic information system. 
20 Numerals 43a to 43g designate pasted images. Numeral 42 
designates a photographic frame of the latest image 43g. 

According to this fourteenth embodiment, even in the case 
where no landmarks are extracted, it is possible to compensate 
superposed display positions, thereby enabling to carry out 
25 the superposed display with high precision, as well as enabling 
to understand situations of a wide range of the ground surface 
having been shot more easily and rapidly. 

Embodiment 15. 

30 The fifteenth embodiment relates to superposed display 
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processing onto the map in the case where plural pieces of images 
are shot continuously in cycles of a predetermined time period 
and a series of plural images are provided as a still image 
data. As for images having been continuously shot, there are 
some images that are subjected to the superposed positional 
compensation by the comparison of landmarks, and other images 
with which the superposed positional compensation by the 
comparison cannot be performed. 

In this case, at the time of real flight, as shown in the 
foregoing fourteenth embodiment, the last compensation values 
continue to be used until the next landmark is extracted. 
However, in the processing of superposed display of an image 
on a map with the use of any image of past flight, a processing 
time period for positional compensation can afford to be spent 
as compared with the case of live flight. Accordingly, in the 
case where the image of past flight is displayed being superposed 
on the map, as shown in Fig. 25, compensation values of each 
parameter that are obtained at a point of land where the next 
landmarks are extracted are applied, going back to the halfway 
point between the point of having executed the compensation 
by landmark comparison last and the current point. 

With reference to Fig. 25, a gray square indicates a landmark 
extraction image, and a white square shows an image from which 
no landmark is extracted. Further, an arrow shows that 
superposed positional compensation values are utilized from 
an image from which landmarks are extracted and with which, the 
superposed positional compensation has been executed, and a 
superposed position is compensated. According to this 
fifteenth embodiment, an overlap state between images in the 
case where any compensation by comparison of landmarks cannot 
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be executed is improved as shown in Fig. 25. 

Fig. 25(a) shows the case where this fifteenth embodiment 
is not applied, and Fig. 25(b) shows the case where this fifteenth 
embodiment is applied. A shot image with which the superposed 
5 display positional compensation of image by the comparison of 
landmarks can be executed is taken as a base point, and the 
layout of images are adjusted back and forth so as to maximize 
the rate of coincidence of overlap parts of images toward the 
halfway point between the shot images with which the superposed 

10 display compensation is executed, whereby the images having 
been continuously shot can be displayed being superposed on 
the GIS map with higher precision. 

According to the fifteenth embodiment, in the processing 
of superposing and displaying the images of past flight on the 

15 GIS map, it is possible to execute the compensation of superposed 
display positions even in the case where no landmark is extracted. 
Furthermore, the overlapping condition between the images is 
not segmented with the image from which a landmark is extracted, 
thus enabling to carry out the superposed display in more smooth 

20 succession with high precision, as well as enabling to understand 
situations of a wide range of the ground surface having been 
shot more easily and rapidly. 

Embodiment 1 6 . 

25 According to this sixteenth embodiment, an altitude 

compensation data of a shot, image to be extracted from flight 
images of the past is linked to a position and registered, whereby 
altitude compensation of a shot point of land is executed even 
in the case where landmarks cannot be extracted from a shot 

30 image. 
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In the case where the altitude compensation processing can 
be executed with the coincidence of landmarks, an altitude 
compensation value obtained as a difference between absolute 
altitude and a relative altitude is registered and managed at 
5 a shot point of land as an altitude compensation value of this 
point, whereby, this altitude compensation value can be utilized 
at any time. Further, in the case where the airframe flies 
at a point of land close to the foregoing point and from the 
next flight on, the altitude compensation can be executed even 
10 at the time of live flight when a processing time period is 
limited, or even in the case where not less than 2 points of 
corresponding landmarks cannot be extracted in a still image 
and a map. 

Fig. 2 6 shows a state in which still images having been 

15 continuously shot are displayed being superposed on the GIS 
map. Explained in this drawing is the case where 2 points of 
landmarks are extracted from the last one piece of image 51 
and the intermediate one piece of image 52, and a compensation 
value of altitude can be obtained. 

20 Not less than 2 points of landmarks are in coincidence with 

the image 51 and the image 52, thereby enabling to obtain a 
compensation value of altitude. When letting these 
compensation values 61 and 62 respectively, the altitude 
compensation values 61 and 62 at points on the map are registered 

25 as symbols. With respect to an image from which not less than. 
2 points of landmarks cannot be extracted, an altitude 
compensation value at this point of land is provided, thus 
executing the compensation of error due to not only a mounting 
angle of the camera but also an altitude of the ground surface, 

30 thereby enabling to superpose and display images having been 



42 



continuously shot on the GIS map with higher precision. 

According to the sixteenth embodiment, by registration of 
an altitude compensation data having been extracted from the 
images of past flight at a point on the map, it is possible 
5 to carry out the altitude compensation with respect to an image 
from which not less than2 points of landmarks cannot be extracted, 
thereby enabling the superposed display with higher precision. 

Industrial Applicability 
10 The present invention is applicable to an image display 

apparatus taking a shot of situations on the ground from on 
board such as helicopter in the case where natural disaster 
such as earthquake or fire occurs or where human disaster such 
as explosion or serious accident occur. 

15 
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